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ABSTRACT 

In  the  fire  safety  community,  the  trend  is  toward  implementing  performance-based  standards  in  place  of  existing 
prescriptive  ones.  Prescriptive  standards  can  be  difficult  to  adapt  to  changing  design  methods,  materials,  and 
application  situations  of  systems  that  ultimately  must  perform  well  in  unwanted  fire  situations.  In  general,  this  trend 
has  produced  positive  results  and  is  embraced  by  the  fire  protection  community.  The  question  arises  as  to  whether 
this  approach  could  be  used  to  advantage  in  cook-off  testing. 

Prescribed  fuel  fire  cook-off  tests  have  been  instigated  because  of  historical  incidents  that  led  to  extensive  damage  to 
structures  and  loss  of  life.  They  are  designed  to  evaluate  the  propensity  for  a  violent  response.  The  prescribed 
protocol  has  several  advantages:  it  can  be  defined  in  terms  of  controllable  parameters  (wind  speed,  fuel  type,  pool 
size,  etc.);  and  it  may  be  conservative  for  a  particular  scenario.  However,  fires  are  inherently  variable  and  prescribed 
tests  are  not  necessarily  representative  of  a  particular  accident  scenario.  Moreover,  prescribed  protocols  are  not 
necessarily  adaptable  and  may  not  be  conservative. 

We  also  consider  performance-based  testing.  This  requires  more  knowledge  and  thought  regarding  not  only  the  fire 
environment,  but  the  behavior  of  the  munitions  themselves.  Sandia  uses  a  performance  based  approach  in  assuring 
the  safe  behavior  of  systems  of  interest  that  contain  energetic  materials.  Sandia  also  conducts  prescriptive  fire 
testing  for  the  IAEA,  NRC  and  the  DOT.  Here  we  comment  on  the  strengths  and  weakness  of  both  approaches  and 
suggest  a  path  forward  should  it  be  desirable  to  pursue  a  performance  based  cook-off  standard. 


INTRODUCTION 

The  goal  of  this  paper  is  to  discuss  the  role  of  testing  &  evaluation  (T&E)  as  a  means  of  demonstrating  that 
requirements  are  being  met.  The  motivation  for  this  paper  comes  from  the  discussions  concerning  fast  cook-off,  both 
with  the  Department  of  Defense  (DoD)  research  community  and  most  recently  with  the  DoD  regulatory  community 
regarding  possible  changes  to  STANAG4240.  The  authors,  from  the  Department  of  Energy  (DOE)  laboratories,  have 
seen  the  role  of  testing  change  significantly  with  the  introduction  of  high  performance  computing  combined  with 
modeling  and  simulation  (M&S)  over  the  last  10  years.  This  combination  is  also  becoming  more  prevalent  within  the 
DoD,  particularly  in  shock  physics,  but  has  not  yet  penetrated  to  demonstrating  cook-off  requirements. 

In  order  to  describe  the  changes  that  occur  with  an  integrated  T&E/M&S  program,  some  terms  need  definitions,  for 
example,  requirements.  The  real  requirement  for  a  set  of  munitions  is  the  performance  of  the  real  systems  in  the  real 
environment.  While  a  motherhood  statement  on  the  surface,  it  in  fact  covers  two  important  aspects  to  the  problem  - 
the  system  response  and  the  environment  in  which  the  system  is  supposed  to  respond  in.  It  is  extremely  important  to 
recognize  that  both  the  system  and  the  environment  are  important  to  meeting  requirements. 

Figure  1  shows  a  simplified  graph  of  how  both  system  performance  and  environmental  loads  show  up.  Obviously,  for 
munitions,  there  are  multiple  environments,  such  as  normal  transportation/storage,  abnormal  accident,  and  hostile 
utilization  environments  and  these  environments  may  include  thermal,  mechanical,  electrical,  electromagnetic,  etc. 
elements.  Similarly,  there  are  multiple  performance  specifications  for  munitions,  and  these  depend  on  environment. 
For  this  paper,  the  discussion  is  limited  to  fast  cook-off  environments  and  hazard  classification  (HC)  and  insensitive 
munitions  (IM)  requirements. 

For  simplicity  of  discussion,  we  will  limit  the  system  performance  discussion  to  Hazard  Division  1.1  vs  Division  1.3 
although  the  discussion  can  be  generalized  as  required. 
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Fig.  1:  Typical  System  Requirements  Include  Both  System  Performance  Metrics  and  Requirements  That  the 

Performance  Be  Met  Under  Environmental  Loads 


One  criterion  in  classification  as  Hazard  Division  1.3  or  lower  (a  system  performance  margin)  is  a  50  ft  fragmentation 
distance  in  a  fast  cook-off  environment.  That  is,  no  metallic  fragments  of  150  g  (0.33  lb)  or  larger  can  be  thrown  a 
distance  of  15  m  (50  ft)  or  more.  Hence,  in  Figure  1  a  test  that  results  in  a  measured  result  in  the  lower  right  quadrant 
is  typically  considered  a  successful  demonstration  of  having  met  system  requirements. 

It  is  generally  accepted  that  system  response  is  a  strong  function  of  the  load;  thus  the  emphasis  on  carefully 
prescribing  the  load  in  terms  of  its  controlling  parameters,  i.e.  parameters  such  as  fuel  type,  fire  size,  duration  and 
location  of  the  test  target  within  the  fire.  This  notion,  while  correct,  is  incomplete.  In  reality,  the  munitions  under  test 
and  the  fire  are  coupled  and  together  behave  as  a  system  that  drives  itself  toward  some  final  outcome.  In  other 
words,  the  munitions  are  not  simply  responding  to  a  fire  input  -  rather  the  munitions  and  the  fire  are  responding 
together  to  a  much  bigger  set  of  controlling  parameters.  This  bigger  set  of  controlling  parameters  includes  the 
characteristics  of  the  local  environment,  the  fire,  and  the  configuration  of  the  munitions  -  or  the  scenario. 

In  the  final  analysis,  the  scenario  determines  the  outcome  of  the  test,  and  is  what  must  be  prescribed.  Such  a 
prescription,  until  recently,  would  have  been  impossible  as  it  requires  detailed  knowledge  of  both  the  fire  environment 
and  the  munitions  in  real  events.  The  available  data  and  correlations  developed  from  cook-off  tests  were  sparse  and 
only  marginally  applicable  to  any  new  test  item.  Even  less  quantifiable  information  was  available  from  actual  events. 
Typical  understanding  of  a  fire  was  that  it  could  be  a  radiating  black  body  at  1000°C  in  certain  situations  and  the 
understanding  of  munitions  was  that  worst  case  occurs  when  heating  uniformly  in  a  heavily  confined  configuration. 
However,  with  the  advent  of  massively  parallel  computing  capability  and  the  accompanying  software,  it  becomes 
possible  to  conduct  high  fidelity  computational  simulations  and  develop  a  more  complete  understanding  of  real-world 
cook-off  events,  which  in  turn  leads  to  designing  a  proper  test  for  any  given  set  of  munitions.  The  process  of 
combining  test  data  and  computation  to  take  scenarios  to  a  final  result  is  termed  performance  based  testing. 

In  what  follows,  a  procedure  for  conducting  performance-based  testing  is  put  forth.  The  underlying  issues  in 
formulating  a  prescriptive  test  are  discussed  both  from  the  point  of  view  of  the  environment  and  from  the  item  under 
test.  It  is  shown  how  a  performance  based  test  overcomes  these  shortcomings.  In  the  performance  based  process,  a 
full  understanding  of  the  thermal  response  of  the  system  is  first  required.  Vulnerabilities  to  heating  need  to  be 
understood,  followed  by  selection  of  scenarios  that  lead  to  such  heating.  This  in  turn  requires  a  full  understanding  of 
the  fire  environment.  The  underlying  tool  for  developing  this  required  understanding  of  both  the  munitions  response 
and  the  fire  environment  is  the  Phenomena  Identification  and  Ranking  Table  (PIRT)  which  establishes  the  gaps  in 
needed  models,  test  data,  and  computational  capabilities.  The  process  of  modeling  and  testing  is  iterative  requiring 
multiple  PIRT  passes  as  unknown  unknowns  are  acquired  and  resolved.  It  is  recognized  that  such  an  elaborate 
undertaking  is  not  always  warranted.  However,  it  is  pointed  out  that  a  single  pass  is  essentially  prescribing  a  test  and 
is  in  fact  the  prescriptive  based  test  approach.  The  subsequent  passes  which  constitute  the  performance  based 
approach  allow  for  continuous  improvement. 


DESCRIBING  THE  ENVIRONMENT  (SEPARATE  FROM  SYSTEM  DESIGN) 


When  an  environment  specification  is  set  independent  of  the  system  design,  the  test  is  usually  a  prescriptive  test. 

For  example,  the  Insensitive  Munitions  (IM)  specification  for  fast  cook-off,  gives  a  prescriptive  test  in  STANAG  4240 
[1].  Paragraph  12  of  STANAG  4240  states  that: 

“The  fuel  fire  tests  are  designed  only  to  simulate  the  most  intense  heating  conditions  likely  to  be 
created  in  a  hydrocarbon  fuel  fire.  They  do  not,  however,  simulate  a  particular  in-service  or 
accident  scenario.”  (emphasis  added) 

Interestingly,  the  Hazard  Classification  (HC)  specification  for  fast  cook-off  testing  at  first  glance  appears  to  take  a 
different  tack.  Instead  of  declaring  a  prescriptive,  conservative  (“most  intense”)  heating  scenario,  the  controlling 
document  (TB-700-2  [2])  states: 

The  rate  of  heating  should  be  credible  in  relation  to  what  might  occur  in  an  accident  during 
transportation,  but  it  is  not  necessary  to  reproduce  precisely  all  the  conditions  of  a  realistic  fire. 

Suitable  methods  include  a  bonfire  using  brushwood,  a  wood  fire  using  a  lattice  of  wooden  laths,  a 
liquid  fuel  fire,  a  propane  burner  and  a  brazier.”  (emphasis  added) 

This  seems  to  imply  a  performance-based  specification,  at  least  in  terms  of  developing  a  “credible”  fire  environment. 
However,  in  practice,  the  two  (IM  and  HC)  specifications  overlap.  In  fact,  the  IM  document  (STANAG  4240)  says  that 
the  procedures  “may  also  be  used  for  Hazard  Classification  (HC)  as  required  by  STANAG  4123.”  This  last  document 
(STANAG  4123  [3])  then  brings  us  full  circle;  for  the  US,  its  implementation  is  via  TB-700-2,  the  HC  document. 

The  approach  of  creating  a  standardized  test  that  is  used  independent  of  the  munitions  being  testing  has  a  long 
tradition  in  testing.  This  basic  approach  has  been  used  for  decades,  not  only  in  DoD  for  munitions,  but  for  other 
hardware.  DOE  used  it  for  decades  for  testing  its  own  hardware  and  the  approach  is  used  extensively  in  the  building 
industry. 

One  of  the  important  outcomes  of  such  testing  is  consistency  and  another  is  low  per-unit  test  cost.  Large  data  bases 
have  been  developed  showing  the  relative  hardware  performance  against  the  test  standard.  The  authors  have  noted 
across  multiple  agencies,  for  multiple  applications,  the  discussions  within  the  communities  often  have  a  distinct 
character.  On  one  hand,  there  is  often  the  admission  that  they  cannot  use  the  test  data  for  any  real  situation  because 
it  does  not  represent  any  real  situation.  On  the  other  hand,  these  data  bases  are  extremely  valuable  because  they 
are  the  only  metric  the  community  has  for  relatively  assessing  hardware  performance.  Across  many  communities,  the 
latter  point  is  felt  so  strongly  that  the  community  cannot  conceive  of  ever  doing  anything  differently  because  of  the 
huge  investment  in  this  approach.  There  is  a  positive  feedback  mechanism  at  work  -  the  more  investment  is  made, 
the  more  reluctant  people  become  to  moving  to  a  different  approach. 

Another  self-propagating  advantage  of  the  prescriptive  testing  approach  is  low  per-unit  test  costs.  Once  testing 
becomes  standardized,  there  is  little  or  no  need  to  introduce  new  hardware  and  personnel  can  become  efficient  in  the 
test  method.  This  results  in  the  lowest  possible  per-unit  test  cost  for  the  development  organization.  Just  as  important, 
when  the  developing  organization  does  its  planning,  it  has  a  known  testing  path  forward  and  can  appropriately  budget 
for  the  testing  required. 

On  the  other  hand,  there  are  some  fundamental  costs  paid  by  continuing  with  this  approach.  Examples  include  the 
logic  used  to  abstract  an  infinite  accident  scenario  space  to  a  finite  number  of  tests  is  essentially  static,  independent 
of  both  weapon  and  platform  changes,  and  the  propagation  of  a  false  sense  of  security  based  on  using  a 
‘conservative  load.'  Documentation  throughout  the  engineering  community  through  the  1960’s  is  so  bad,  that  often 
times,  the  original  logic  used  to  abstract  an  infinite  accident  space  to  a  finite  number  of  tests  is  no  longer  accessible. 
It  is  easy  to  understand  in  a  world  where  steel  &  aluminum  are  the  primary  materials  used  for  platform  construction, 
why  a  liquid  hydrocarbon  fire  standard  is  chosen.  However,  as  more  and  more  platforms  acquire  more  and  more  fuel 
from  their  composite  structures,  how  is  it  that  this  change  is  fundamentally  reflected  in  the  test  standards? 

Perhaps  the  most  pernicious  problem  with  standardized  testing  that  is  independent  of  system  performance  is  the 
concept  of  “conservative.”  Everyone  agrees  that  for  safety  testing,  that  the  test  should  be  conservative.  However, 
there  is  a  difference  between  measuring  a  conservative  system  response,  providing  a  conservative  system  load,  and 
using  a  conservative  scenario.  As  noted  in  the  introduction,  the  real  requirement  is  the  performance  of  the  real 


*  The  details  of  STANAG  4240,  and  the  specifics  around  changes  to  the  environment  to  permit  propane 
testing  are  not  relevant  to  this  discussion. 


system  in  the  real  environment.  When  tests  are  standardized  independent  of  the  system  design,  the  result  of  the  test 
cannot  be  a  conservative  demonstration  of  the  system  response. 


Only  in  a  very  simplistic  case  can  an  ‘overtest’  result  in  a  conservative  response.  Take  a  simple  mechanical  spring 
analogy  for  example.  If  a  system  has  a  single,  one-degree-of-freedom  response  mode,  then  an  overtest,  e.g.,  the 
application  of  a  load  beyond  what  is  expected  in  the  environment  will  result  in  a  conservative  response  of  that  spring. 
However,  if  there  are  multiple  modes,  for  example,  multiple  springs,  then  the  same  overtest  load  cannot  be  said  to  be 
conservative  because  the  partitioning  of  the  energy  between  response  modes  that  may  have  different  susceptibilities 
to  failure.  Such  testing  does  not  result  in  a  conservative  design.  It  is  simply  a  design  that  has  been  tested  to  the 
wrong  requirements.  Since  munitions  contain  multiple  response  modes,  a  standardized  test  cannot  possibly  be 
considered  to  result  in  a  conservative  response.  Hence  the  common  overheard  conversation  that  the  test  results  are 
important  in  a  relative  sense,  because  they  cannot  be  applied  to  real  world  situations  with  any  confidence. 

If  a  standardized  test  does  not  result  in  a  conservative  measure  of  the  design,  then  it  can  either  represent  a 
conservative  system  load  or  a  conservative  scenario,  or  more  often  than  not,  neither.  Early  in  the  authors  careers  it 
was  commonly  thought  that  putting  a  weapon  a  meter  above  a  fully  engulfing  liquid  hydrocarbon  pool  fire  would  result 
in  a  conservative  system  load,  i.e.,  that  would  be  the  hottest  place  in  a  fire.  Testing  throughout  the  1990’s  showed 
that  this  was  not  in  fact  the  case.  All  pool  fires,  or  jet  flames  for  that  matter,  have  a  fuel  vapor  core  that  is  significantly 
cooler  than  the  turbulent  flame  brush.  Perhaps  not  surprisingly,  the  fire  is  hottest  where  the  fuel  and  air  are  burning 
most  vigorously,  which  in  a  turbulent  diffusion  flame,  is  near  the  stoichiometric  surface.  For  heavy  hydrocarbon  fuels, 
this  is  where  relatively  larger  amounts  of  air  are  entrained,  i.e.,  near  the  edge  of  the  fire. 

The  use  of  the  word  conservative  relative  to  a  scenario  is  usually  thought  of  as  meaning  of  “low  probability.”  If 
quantifiable,  and  used  within  the  context  of  probabilistic  risk  assessment  methodologies,  it  has  meaning.  Otherwise,  it 
is  very  often  misunderstood  to  mean  a  scenario  resulting  in  a  conservative  load. 

SYSTEM  PERFORMANCE  SPECIFICATION 

A  complete  system  performance  specification  suffers  because  the  system  response  modes  are  often  not  known.  In 
fact,  one  could  say,  this  lack  of  knowledge  is  the  very  reason  for  doing  the  prescribed  test.  However,  we  point  out  that 
the  single  prescribed  test  uncovers  only  one  mode.  Furthermore,  the  current  test  strategy  with  its  go/no-go  results 
give  no  insight  into  physics  behind  system  performance.  This  results  in  a  guarantee  of  testing  for  eternity  as  test  data 
cannot  be  extrapolated  to  new  designs.  A  proposed  approach  is  the  PIRT  process  which  seeks  to  identify  the 
important  physics. 

As  an  example,  the  PIRT  process  was  used  in  a  workshop  held  at  the  May  2008  JANNAF  meeting.  The  workshop 
was  set  up  to  address  the  issue  of  the  technical  problems  associated  with  cook-off  of  large  rocket  motors  in  fire 
environments.  Forty-seven  individuals,  representing  all  branches  of  the  US  Department  of  Defense,  Department  of 
Energy,  industry,  academia  and  knowledgeable  consultants  participated  in  the  workshop.  The  first  step  of  the 
process  was  the  specification  of  a  credible  scenario.  Here  we  chose  to  investigate  the  scenario  of  an  accident 
involving  a  large  rocket  motor  during  over-the-road  transport.  Physical  phenomena  associated  with  the  accident 
event,  the  resulting  fire,  and  the  potential  responses  of  the  motor  were  identified  by  participants  during  a  facilitated 
brain-storming  session.  The  individual  phenomena  were  then  ranked  by  the  group  with  respect  to  their  perceived 
importance  to  the  final  outcome  of  interest  -  the  throwing  of  fragments  from  the  motor.  The  adequacy  of 
experimental  data  and  models  to  describe  these  phenomena  were  also  ranked  by  meeting  participants.  The  end 
result  of  the  workshop  was  a  prioritized  list  or  table  (the  PIRT)  of  important  physical  phenomena  for  which  there 
inadequate  data  and/or  models  to  allow  for  prediction  of  the  cook-off  response.  It  was  hoped  that  this  exercise  and 
the  PIRT  would  be  useful  as  a  guide  to  prioritizing  future  research  in  the  field.  The  workshop  report  was  recently 
published  as  an  unlimited  release  document.  [4] 

COMBINING  T&E/M&S 

Using  a  combination  of  testing  &  evaluation  and  modeling  &  simulation  is  nothing  more  than  using  the  scientific 
method.  A  physics-based  numerical/computational  model  is  nothing  more  than  the  codification  of  theory.  A  simulation 
based  on  the  numerical/computational  model,  or  code  as  it  is  often  called,  is  nothing  more  than  a  hypothesis.  Testing 
takes  on  its  usual  role  -  truth  is  always  established  empirically.  Thus  combining  T&E  and  M&S  is  the  application  of 
the  scientific  method  to  product  testing. 

Combining  T&E/M&S  does  not  change  the  purpose  of  testing:  that  is  to  build  up  a  body  of  evidence  that  supports  a 
decision  that  the  hardware  in  question  meets  its  requirements.  What  does  change  when  combining  M&S  with  T&E  is 
the  methodology,  the  quality  of  the  body  of  evidence,  and  the  costs.  The  methodology  is  iterative.  First  simulations  of 


the  proposed  experiments  are  run  to  evaluate  the  evidence  that  will  be  obtained  by  running  the  experiment. 
Specifically,  the  simulations  will  evaluate  whether  the  test  will  optimally  produce  the  body  of  evidence  desired.  All 
aspects  of  the  test  design  can  be  optimized  with  in  the  uncertainty  of  the  simulation  predictions.  Specifically  checked 
is  whether  the  instrumentation  will  remain  in  range  and  whether  the  output  from  that  instrumentation  will  confirm  the 
desired  results. 

The  resulting  test  will  not  only  provide  the  body  of  evidence  that  confirms  the  hardware,  but  can  also  be  used  to 
validate  the  model.  By  validate  we  mean  establish  the  uncertainty  in  the  prediction  of  the  model.  To  accomplish  the 
latter,  it  is  necessary  to  include  extra  diagnostics  to  measure  the  uncertainty  in  the  boundary  conditions  that  are  not 
normally  needed  for  pure  standardized  tests.  The  test  can  be  thought  of  in  the  scientific  sense  of  a  search  for  the 
unknown  unknown  not  contained  within  the  theory.  If  the  test  result  is  contained  within  the  uncertainty  of  the 
theoretical  prediction,  then  the  test  (with  the  extra  diagnostics)  can  be  used  to  reduce  the  bands  of  prediction 
uncertainty. 

In  this  regard,  M&S  and  T&E  methods  are  synergistic.  T&E  can  be  thought  of  as  the  whole  truth,  partially  exposed, 
while  M&S  can  be  thought  of  as  a  partial  truth,  wholly  exposed.  T&E  is  limited  primarily  by  diagnostics  that  can 
extract  useful  physics  insight  into  the  test,  while  M&S  is  primarily  limited  by  the  amount  of  physics  that  can  be 
contained  within  the  model  and  still  solved.  T&E’s  primary  strength  is  the  physics  content  within  the  test  while  M&S 
strength  is  in  the  diagnostics  that  can  be  used  to  probe  the  physics  within  the  model.  For  developing  the  evidence 
base  for  hardware  qualification  (i.e. ,  it  meets  requirements),  both  M&S  and  T&E  elements  are  aided  by  application  of 
statistical  mathematics  to  appropriately  combine  uncertainties  into  quantitative  metrics  defining  the  relative 
confidence  one  has  due  to  the  inherent  uncertainties  in  the  combined  methodology. 

The  combined  approach  is  necessary  to  address  two  of  the  shortfalls  found  in  prescriptive  testing,  which  is  how  to 
incorporate  real  scenarios  and  how  to  incorporate  system  response.  Since  abnormal  or  safety  environments  are 
unbounded,  it  is  not  possible  by  test  alone  to  address  how  the  environment  will  challenge  the  system.  Through  M&S  it 
is  possible  to  run  any  specific  scenario  of  concern  and  address  the  relative  difference  in  the  thermal  loads  on  the 
system.  In  this  way,  after  acceptance,  a  means  exists  such  that  “what  if”  studies  can  be  done  to  address  scenarios 
that  were  not  considered  in  the  design  of  the  prescriptive  test.  Further,  with  M&S,  it  is  possible  to  assess  system 
performance  margins,  not  just  go/no  go  acceptance  criteria,  as  will  be  developed  in  the  next  section. 

Of  course,  all  this  added  benefit,  comes  at  a  price.  In  the  short  run,  the  costs  are  in  fact  higher  but  in  the  long  run  they 
are  expected  to  be  much  lower.  For  a  single  development  program,  the  cost  of  demonstrating  that  requirements  are 
being  met  will  more  than  double  by  a  combined  T&E  and  M&S  approach.  This  point  is  easily  seen  in  that  T&E 
personnel  &  equipment  costs  are  fundamentally  different  than  M&S  personnel  &  equipment  costs.  In  addition  to  the 
T&E  costs,  which  are  not  reduced  but  increased,  one  needs  trained  computational  personnel  and  high  performance 
computers.  To  lead  order,  the  cost  of  M&S  is  on  the  same  order  as  T&E  when  all  hardware  costs  are  taken  into 
account.  Further,  as  noted  above,  T&E  costs  increase  due  to  increased  instrumentation  required  to  validate  M&S 
results. 

If  the  per-unit  testing  cost  is  higher,  how  is  it  possible  for  the  combined  T&E/M&S  approach  to  result  in  lower  overall 
costs?  The  answer  lies  in  the  fact  that  demonstrating  that  a  product  meets  requirements  is  only  part  of  the 
design/development  cycle,  and  through  the  combined  approach,  the  whole  cycle  cost,  over  time,  can  be  decreased 
significantly.  This  point  is  not  intuitively  obvious  and  needs  explanation  with  examples. 

As  noted  above  the  combined  T&E/M&S  approach  is  nothing  more  than  the  application  of  the  scientific  method  and, 
over  the  long  haul,  nothing  has  proven  more  successful  at  driving  technology  forward  than  the  application  of  the 
scientific  method.  A  well  understood  example  exists  within  the  automotive  manufacturing  community.  Arguably,  the 
US  had  the  most  experience  in  manufacturing  of  complex  systems  such  as  automobiles  of  anybody  in  the  world  prior 
to  the  1980’s.  After  all,  the  US  had  pioneered  assembly  line  manufacturing.  And  yet  within  a  decade  or  two,  the  entire 
process  was  revolutionalized  by  the  Japanese  application  of  Deming’s  methods.  Simply  put,  the  method  was 
basically  to  instrument  the  manufacturing  process  to  reduce  the  number  of  products  not  passing  requirements.  By 
understanding  the  process,  it  could  be  improved  and  over  a  number  of  cycles,  result  in  a  lower  overall  cost  with  a 
significantly  improved  product  quality.  In  essence,  testing  took  on  an  additional  role.  Instead  of  just  testing  the 
product,  and  throwing  away  defective  product,  testing  of  the  process  improved  the  manufacturing  process. 

This  is  in  essence  what  the  authors  are  proposing  will  happen  to  the  design  process.  Through  a  combination  of 
T&E/M&S  with  an  expanded  role  for  M&S  to  include  testing  (validation)  of  the  physics  models  (M&S)  on  which  design 
is  based,  the  design  process  will  become  more  productive  and  result  in  lower  costs.  As  a  specific  example,  take  the 
formulation  of  new  propellants  for  large  rocket  motors.  This  was  the  subject  of  the  PIRT  exercise  explained  in  system 
performance  specification  section.  By  the  time  large  rocket  motors  are  tested,  a  failure  to  make  Class  1.3  can  be  very 
expensive.  If  the  existing  product  testing  T&E  community  could  be  utilized  not  just  to  determine  pass/fail  at  the  end 


stage  of  development,  but  can  be  leveraged  to  understand  the  design  process  at  an  early  stage,  then  considerable 
savings  would  result  as  well  as  a  vastly  improved  range  of  products.  The  authors  believe  this  is  the  design  equivalent 
of  the  manufacturing  advance  made  in  the  1980-2000  time  frame. 

DESCRIBING  THE  ENVIRONMENT  &  SYSTEM  DESIGN  ITERATIVELY 

To  revolutionize  the  design  process  with  a  combined  T&E/M&S  approach,  it  is  necessary  to  incorporate  the  design 
into  testing,  i.e.,  to  move  beyond  prescriptive  tests  to  performance  based  testing.  As  stated  above,  the  real 
requirement  is  the  performance  of  the  real  system  in  the  real  environment.  Thus,  the  two  attributes  of  system 
performance  and  the  environmental  loads  shown  in  Figure  1  cannot  be  separated.  However,  the  combination 
presents  a  subtle  “chicken  and  egg”  type  difficulty.  One  cannot  know  system  performance  until  there  is  a  system 
design,  and  one  cannot  design  a  system  to  perform  appropriately  until  the  load  is  understood. 

The  current  approach  within  the  DOE  community  for  abnormal  accident  environments  for  which  there  are  weapon 
safety  performance  requirements  is  to  adopt  an  iterative  approach,  which  has  been  termed  representative, 
conservative,  accident  scenario  (RCAS).  For  a  given  accident  scenario,  the  twin  adjectives,  representative  and 
conservative  apply  to  the  environment  and  system  response  respectively.  Scenarios  are  chosen  to  represent  a  broad 
class  of  conditions  in  which  the  weapon  may  find  itself.  Not  surprisingly,  large  hydrocarbon  pool  fires  represent  one 
class  of  accidents.  However,  the  second  adjective,  conservative,  changes  the  nature  of  the  test  from  prescriptive  to 
performance  based.  Conservative  means  to  use  unknowable  load  details  within  the  representative  scenario  to  drive  a 
system  design  implementation  vulnerability.  In  this  way,  the  design  and  the  environment  are  coupled.  By  necessity, 
as  the  design  evolves,  the  test  will  evolve  as  well. 

For  this  process  to  end  up  being  cost  effective,  the  validation  part,  i.e.,  establishing  the  uncertainty  in  the  numerical 
simulation  tools,  must  occur  early  enough  that  the  simulation  tools  can  be  utilized  early  in  the  design  cycle.  It  is 
through  the  simulation  tool  application  in  design  as  well  as  development  testing,  that  the  design  process  becomes 
“instrumented”  in  the  manner  that  was  described  for  manufacturing  processes.  Rather  than  simply  conducting  go/no- 
go  testing  and  throwing  parts  away,  the  T&E  community  can  become  more  effective  during  the  design  process, 
resulting  in  cost  savings. 


THE  CHALLENGE  FOR  DOD  FAST  COOK-OFF  TESTING 

DOE  has  been  able  to  adopt  the  performance  approach  because  of  the  limited  types  of  munitions  in  its  purview  and 
the  potential  high  consequences  involved.  Does  the  same  apply  to  the  DoD  with  its  multitude  of  munitions?  While 
prescriptive  testing  is  cheap(est)  on  a  per  test  basis,  the  fact  that  progress  is  glacial  means  that  overall  it  is  more 
expensive  than  a  combined  T&E  and  M&S  approach.  So  what  is  the  challenge? 

The  primary  challenge  to  the  DoD  community  to  implement  this  approach  for  fast  cook-off  testing  is,  in  the  authors’ 
opinion,  primarily  one  of  being  able  to  physically  describe  the  system  response  mode.  In  other  words,  what 
mechanisms,  or  series  of  mechanisms  within  the  system  results  in  fragments  being  thrown  over  50  ft  when  heated  in 
a  fire.  All  the  other  major  challenges  are  being  worked  in  parallel  through  other  programs.  If  DoD  wants  a  combined 
approach,  then  it  must  invest  in  understanding  the  physics/chemistry  of  system  response.  DDESB  must  be  the  driver 
for  this  understanding.  We  encourage  the  DDESB  to  continue  to  strongly  support  the  R&D  community  to  move  this 
understanding  forward. 
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Prescriptive  Test  Approach 


Performance  Based  Process 


•Poorly  understood  phenomena  are  isolated. 

•Use  specified  test  to  control  the  poorly  understood 
phenomena  (e.g.  declare  a  specification  and  be 
consistent) 

•Hope  is  for  consistent  tests  to  allow  comparison 
(but  are  they  Representative?  Conservative?) 
•Defined  procedure  allows  a  priori  estimation  of  test 
program  costs;  cheaper  for  single  program 


•Poorly  understood  phenomena  are  flagged  for 
further  investigation 

•Representative  scenario  for  system  of  interest 
•Combined  experimental  &  modeling 

•  Series  of  experiments  and  models  highlighting 
specific  physics 

•  Builds  up  toward  qualification  test  &  simulations 
•Simulation-aided  design  allows  long-term  cost 
savings  (costs  not  reduced  for  a  single  system) 


Broader  foundation  of  Experience  results  from  performance  based  process. 


ombining  Test/Evaluation  With 
Modeling  and  Simulation 


*  Using  a  combination  of  testing  &  evaluation  and 
modeling  &  simulation  is  nothing  more  than 
using  the  scientific  method. 

*  A  physics-based  numerical/computational  model 
is  nothing  more  than  the  codification  of  theory. 

*  A  simulation  based  on  the 
numerical/computational  model,  or  code  as  it  is 
often  called,  is  nothing  more  than  a  hypothesis. 

*  Testing  takes  on  its  usual  role  -  truth  is  always 
established  empirically. 

Combining  T&E  and  M&S  is  the  application  of 
the  scientific  method  to  product  testing. 
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information  come  from? 
Are  the  models  as  being 
used  valid? 


•  Detailed  Heat  Transfer 
Paths 

•  Decomposition 
Chemistry  of  all 
materials 

•  Post  Ignition  behavior 

Prescriptive  challenges  can  be  difficult  if  not  impossible  to  address 
adequately. 


•  The  broader  testing  community  is  already  addressing  performance 
based  challenges  for  the  thermal  load. 


•  Understanding  munitions  response  is  the  critical  path  to  move  to 
performance-based  process. _ 

We  encourage  the  DDESB  to  continue  to  strongly  support  the 
R&D  community  to  enable  the  performance-based  process. 


®Sandia 
National 
Laboratories 


Acknowledgements 


•This  work  was  supported  by  Sandia  National  Laboratories  under  the 
Campaign  6  and  Weapons  Systems  Qualification  programs. 

•Sandia  is  a  multiprogram  laboratory  operated  by  Sandia  Corporation,  a 
Lockheed  Martin  Company,  for  the  United  States  Department  of 
Energy’s  National  Nuclear  Security  Administration  under  Contract  DE- 
AC04-94AL85000 


Sandia 

National 

Laboratories 


